Cs-621 Theory Gems 1 Learning Non-linear Classifiers

In the previous lectures, we have focused on finding linear classifiers, i.e., ones in which the decision boundary is a hyperplane. However, in many scenarios the data points cannot be really classified in this manner, as there simply might be no hyperplane that separates most of the positive examples from the negative ones see, e.g., Figure 1 (a). Clearly, in such situations one needs to resort to more complex (non-linear) classifiers and thus one would expect that there is no use here for the linear classification algorithms we developed so far. Fortunately, as we will see in this lecture, this is not really the case as there actually are powerful and convenient ways of performing a non-linear classification by building on the algorithms for the linear one. In particular, we will see two very useful and quite broadly-applicable techniques: the Kernel Trick (or just kernelization) and boosting.